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1. (AMENDED) An image coding apparatus^ comprising: 

inputting means for inputting a multiplexed stream 

containing multimedia coding data; 

separating means for separating a video stream from 

said input multiplexed stream input by said inputting means ; 

converting means for performing a predetermined 

conver sion ting process on said separated video stream 

separated — fey — said — separating — mcans to form a converted video 

stream ; 

generating means for generating additional 
information indicating that a mismatch will occur when the 
said converted video stream is displayed on the basis of said 
multimedia coding data; and 

outputting means for outputting said converted video 
stream, said multimedia coding data, and said additional 
information. 

5. (AMENDED) The image coding apparatus according to 
claim 1, wherein the — conversion — by — said conver sion ting means 
process includes at least a process of decoding the — said 
separated video stream separated — by — said — separating moans — and 
a process of encoding the said decoded video stream. 

6. (AMENDED) The image coding apparatus according to 
claim 1, wherein said additional information generated by said 
generating — means — contains at least one of original picture 
frame information and an original screen aspect ratio. 

7. (AMENDED) The image coding apparatus according to 
claim 1, wherein said additional information generated by said 
generating means contains an original video format and a video 
format after the said conversion process . 

8. (AMENDED) The image coding apparatus according to 
claim 1, wherein said additional information generated by said 
generating means contains an original screen aspect ratio and 
a screen aspect ratio after the said conversion process . 

9. (AMENDED) The image coding apparatus according to 
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claim 1, wherein said additional information generated by said 

generating means contains at least one of information 

indicating whether or not a picture frame of said video stream 
has been converted by said converting means, information about 
an original picture frame of said separated video stream 
separated — by — said — separating — means , and an original screen 
aspect ratio. 

10. (AMENDED) An method for image coding^ method 
comprising thc stops of : 

inputting a multiplexed stream containing multimedia 
coding data; 

separating a video stream from said input 
multiplexed stream y input in said inputting step; 

performing a predetermined conver sion ting process on 
said separated video stream separated — in — said — separating 
■s^ep to form a converted video stream ; 

generating additional information indicating that a 
mismatch will occur when &Fte — said converted video stream is 
displayed on the basis of said multimedia coding data; and 

outputting said converted video stream, said 
multimedia coding data, and said additional information. 

11. (AMENDED) The image coding method according to claim 
10, further comprising the step of : 

coding said additional information generated in said 
generating — step as data separate from said multiplexed stream 
containing said converted video stream. 

12. (AMENDED) The image coding method according to claim 
10, further comprising: 

a coding step e£ multiplexing said additional 

information generated in said generating step with said 

multiplexed stream containing said converted video stream and 
then coding a multiplexed result. 

13. (AMENDED) The image coding method according to claim 
10, wherein said conversion process includes converting a 
video stream picture frame parameter — is — converted — in — said 
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converting step . 

14. (AMENDED) The image coding method according to claim 
10, wherein the said conversion processi n — said converting step 
includes at least a process of decoding ^fcke — said separated 
video stream separated — in — said — separating — step — and a process 
of encoding the said decoded video stream. 

15. (AMENDED) The image coding method according to claim 
10, wherein said additional information generated — in — said 
generating — stop contains at least one of original picture 
frame information and an original screen aspect ratio. 

16. (AMENDED) The image coding method according to claim 
10, wherein said additional information generated — in — said 
generating — step contains an original video format and a video 
format after the said conversion process . 

17. (AMENDED) The image coding method according to claim 
10, wherein said additional information generated — in — said 
generating — stop contains an original screen aspect ratio and a 
screen aspect ratio after the said conversion process . 

18. (AMENDED) The image coding method according to claim 
10, wherein said additional information generated — in — said 

generating step contains at least one of information 

indicating whether ef — not a picture frame of said video stream 
has been converted i nwith said conver sion ting step process , 
information about an original picture frame of said separated 

video stream^ separated — in — said — separating step, and an 

original screen aspect ratio. 

19. (AMENDED) A recording medium recorded with a 
computer-readable program for image coding , said program 
comprising the steps of : 

inputting a multiplexed stream containing multimedia 
coding data; 

separating a video stream from said input 
multiplexed strea m input in said inputting stop ; 

performing a predetermined conver sion ting process on 
said separated video stream separated — in — said — separating 
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etepto form a converted video stream ; 

generating additional information indicating that a 
mismatch will occur when the — said converted video stream is 
displayed on the basis of said multimedia coding data; and 

outputting said converted video stream, said 
multimedia coding data, and said additional information. 

20. (AMENDED) An information recording medium recorded 
comprising : 

a data recording area recorded with a video stream 
converted by a predetermined conversion process, multimedia 
coding data, and additional information indicating an 
occurrence of a mismatch at — when displaying said converted 
video stream on the basis of said multimedia coding data. 

26. (AMENDED) The recording medium according to claim 
20, wherein said additional information contains information 
about an original video format and information about a video 
format after the said conversion process . 

27. (AMENDED) The recording medium according to claim 
20, wherein said additional information contains information 
about an original screen aspect ratio and information about a 
screen aspect ratio after the said conversion process . 

28. (AMENDED) The recording medium according to claim 
20, wherein said additional information contains at least one 
of information indicating whether — net — a picture frame of 
said video stream has been converted, information about an 
original picture frame of said video stream, and information 
about an original screen aspect ratio. 

29. (AMENDED) An image decoding apparatus^ comprising: 
inputting means for inputting a multiplexed stream 

containing multimedia coding data; 

separating means for separating a video stream from 
said input m ultiplexed stream input by said inputting means ; 

decoding means for decoding said separated video 
strea m separated by oaid separating means ; and 

processing means for performing a predetermined 
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conversion process on said decoded video stream in accordance 
with additional information indicating an occurrence of a 
mismatch at — when displaying said decoded video stream on the 
basis of said multimedia decoding data. 

30. (AMENDED) The image decoding apparatus according to 
claim 29, further comprising: 

acquiring means for acquiring said additional 
information from data different from — than said multiplexed 
stream. 

31. (AMENDED) The image decoding apparatus according to 
claim 29, further comprising: 

acquiring means for acquiring said additional 
information from a multiplexed stream with — which — containing 
said additional information is multiplexed . 

33. (AMENDED) The image decoding apparatus according to 
claim 29, wherein the conversion processb y — said — processing 
mcano includes at least a process of decoding said separated 
video stream separated by — said — separating means — and a process 
of encoding the said decoded video stream. 

35. (AMENDED) The image decoding apparatus according to 
claim 29, wherein said additional information contains an 
original video format and information about a video format 
after the said conversion process . 

36. (AMENDED) The image decoding apparatus according to 
claim 29, wherein said additional information contains 
information about an original screen aspect ratio and 

information about a screen aspect ratio after t-he said 

conversion process . 

37. (AMENDED) The image decoding apparatus according to 
claim 29, wherein said additional information contains at 
least one of information indicating whether er — net — a picture 
frame of said video stream has been converted by said 
converting — processing m eans, information about an original 
picture frame of said separated video strea m separated by said 
separating — means , and information about an original screen 
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aspect ratio. 

38. (AMENDED) Aft method for image decoding^ method 
comprising the — steps of : 

inputting a multiplexed stream containing multimedia 
coding data; 

separating a video stream from said input 
multiplexed strea m input in said inputting step ; 

decoding said separated video stream — separated — in 
said separating stop ; and 

performing a predetermined conversion process on 
said decoded video stream in accordance with additional 
information indicating an occurrence of a mismatch a^t — when 
displaying said decoded video stream on the basis of said 
multimedia decoding data. 

39. (AMENDED) The image decoding method according to 
claim 38 , further comprising the step of : 

acquiring said additional information from data 
different from than said multiplexed stream. 

40. (AMENDED) The image decoding method according to 
claim 38 , further comprising the stop of : 

acquiring said additional information from a 

multiplexed stream with which — containing said additional 

information is multiplexed . 

41. (AMENDED) The image decoding method according to 
claim 38, wherein said conversion process includes converting 
a picture frame parameter of said video stream is — converted in 
said processing step . 

42. (AMENDED) The image decoding method according to 

claim 38, wherein the said conversion processift said 

processing — stop — includes at least a process of decoding said 
separated video stream separated in said separating — step and a 
process of encoding the said decoded video stream. 

46. (AMENDED) The image decoding method according to 
claim 38, wherein said additional information contains at 
least one of information indicating whether — n^fe — a picture 
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frame of said video stream has been converted in — with said 
conver sion ting process otcp , information about an original 
picture frame of said separated video strea m ocparatcd in — oaid 
ocparating — step, and information about an original screen 
aspect ratio . 

47. (AMENDED) A recording medium recorded with a 
computer-readable progra m for image decoding , said program 
comprising the otcpo of : 

inputting a multiplexed stream containing multimedia 
coding data; 

separating a video stream from said input 
multiplexed stream input — in oaid inputting — otcp ; 

decoding said separated video stream — separated — in 
said separating step ; and 

performing a predetermined conversion process on 
said decoded video stream in accordance with additional 

information indicating occurrence of a mismatch efe when 

displaying said decoded video stream on the basis of said 
multimedia decoding data. 

48. (AMENDED) An image coding apparatus^ comprising: 
inputting means for inputting a multiplexed stream; 
separating means for separating a video stream from 

said input m ultiplexed stream input by oaid inputting means ; 

determining means for determining whether — nefe 
multimedia coding data is contained in said input multiplexed 
strea m input by oaid inputting mcano ; 

generating means 7 i£ it ±-s determined — by oaid 

determining mcano that oaid multimedia coding data is- 

contained — in — oaid — multiplexed — stream, for generating coding 
control information based on the presence of said multimedia 
coding data in said multiplexed stream for instructing net — fee 
change — that a display format of said separated video stream 
ocparatcd by oaid ocparating mcano not be changed ; 

converting means for performing a predetermined 
conversion process on said separated video stream ocparatcd by 

16 



Application No. 09/872,147 



said — separating — moans — on the basis of said coding control 

information generated — by said — generating — mcans to form a 

converted video stream ; and 

multiplexing means for generating a multiplexed 

stream which that contains said converted video stream 

converted by said converting means . 

49. (AMENDED) The image coding apparatus according to 

claim 48, wherein said generating means coding control 

information provides instruct ion s that ft efe — fee — change — any of a 
picture frame, a video format, and an aspect ratio of said 
separated video stream not be changed . 

50. (AMENDED) Aft method for image coding^ method 
comprising the steps — e#: 

inputting a multiplexed stream; 

separating a video stream from said input 
multiplexed stream input in said inputting stop ; 

determining whether or not m ultimedia coding data is 
contained in said input m ultiplexed stream — input — in — said 
inputting stop ; 

generating coding control information based on the 
presence of said multimedia coding data in said multiplexed 
stream for instructing ftefe — fee — change — that a display format of 
said separated video stream not be changed scparatod — in — said 
separating — step, — i£ — ife — i-s — determined in — said determining — step 

that said multimedia coding data i-s contained if* said 

multiplexed stream ; 

performing a predetermined conversion process on 
said separated video stream separated — ift — said — separating — step 
on the basis of said coding control information to form a 
converted video stream gonerated in said generating stop ; and 

generating a multiplexed stream which that contains 
said converted video stream converted in said converting step . 

51. (AMENDED) The image coding method according to claim 
50, wherein said coding control information — provides 
instruct ionsiftg not — fee — change — afty — ef — that a picture frame, a 
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video format, and an aspect ratio is generated in feke 

generating step of said separated video stream not be changed . 

52. (AMENDED) A recording medium recorded with a 
computer-readable progra m for image coding , said program 
comprising the stops of : 

inputting a multiplexed stream; 

separating a video stream from said input 
multiplexed strea m input in said inputting stop ; 

determining whether or not multimedia coding data is 
contained in said input m ultiplexed stream — input — in — said 
inputting stop ; 

generating coding control information based on the 
presence of said multimedia coding data in said multiplexed 
stream for instructing net — fee — change that a display format of 
said separated video stream not be changed scparatcd — in — said 
separating — step, — if — ife — is — determined — in — said — determining — step 

that said multimedia coding data is contained in said 

multiplexed stream ; 

performing a predetermined conversion process on 
said separated video stream separated — in — said — separating — stop 
on the basis of said coding control information to form a 
converted video stream genoratod in said generating step ; and 

generating a multiplexed stream which — that contains 
said converted video strea m converted in said converting step . 

53. (AMENDED) An information recording medium^ 

comprising : recorded 

a data recording area recorded with coding control 
information for instructing nefe — tre — change — that a display 
format of not be changed for a video stream and a multiplexed 
stream containing a video stream on which a predetermined 
conversion process has been performed on the basis of said 
coding control information. 
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REMARKS 

The above-noted amendments of claims 1, 5-20, 26-31, 
33, 35-42, and 46-53, as well as the submission of a revised 
Abstract and revisions to the Specification, are respectfully 
submitted prior to initiation of the prosecution of this 
application in the U.S. Patent and Trademark Office. 

The above-noted amendments to the claims are 
respectfully submitted in order to more clearly and 
appropriately claim the subject matter which applicant 
considers to constitute his inventive contribution. None of 
these amendments are intended to narrow the scope of any of 
the claims, and no new matter is included therein. In 
addition, the revisions to the Abstract and Specification are 
submitted in order to clarify and correct the Abstract and 
Specification and to conform them to all of the requirements 
of U.S. practice. No new matter is included in these 
amendments . 

In view of the above, it is respectfully requested 
that these amendments now be entered, and that prosecution on 
the merits of this application now be initiated. If, however, 
for any reason the Examiner does not believe such action can 
be taken, it is respectfully requested that he/she telephone 
applicant's attorney at (908) 654-5000 in order to overcome 
any objections which he/she may have. 
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If there are any additional charges in connection 



with this requested amendment, the Examiner is authorized to 
charge Deposit Account No. 12-1095 therefor. 



600 South Avenue West 
Westfield, NJ 07090-1497 
Telephone: (908) 654-5000 
Facsimile: (908) 654-7866 

348623_1.DOC 



Respectfully submitted, 




LERNER, DAVID, LITTENBERG, 
^-KRUMHOLZ & MENTLIK, LLP 



Robert b. cohen 

Reg. No. 32,768 
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ABSTRACT OF THE DISCLOSURE 

JjMDfcl*!^ This invention is intended to prevent a mismatch 
from occurring between a re-encoded and recorded video 
stream and other data. A transport stream is input to-ana a 
recording apparatus and is then supplied to a demultiplexer 
and an analyzing block. The analyzing block outputs non- 
video stream information to a coding controller. The coding 
controller controls the encoding operation of an encoder on 
the basis of the input information and outputs video coding 
control information to a coding block. On the basis of 
video coding control information, the coding block generates 
multimedia display sub-information (including information 
about a picture frame, for example) which prevents a 
mismatch from occurring on the reproducing side between 
video and other data — eR — the — reproducing — side , and outputs 
this information to a writing block. The writing block 
records the video stream and other data to a recording 
medium^ as well as recording the multimedia display sub- 
information generated by the coding block. 
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J? RECEIVED 

AND METHOD FOR IMAGE CODING AND DECODING ^ 

JUfT2 0 2002 

CROSS-REFERENCE TO RELATED APPLICATIONS - . . 

Technology Center 2600 

[0001] The present application claims priority from 
Japanese Application Nos. P2000-165298 filed June 2, 2000 
and P2001-001031 filed January 9, 2001, the disclosures of 
which are hereby incorporated by reference herein. 
BACKGROUND OF THE INVENTION 

[0002] The present invention relates generally to an 
image coding apparatus and method, an image decoding 
apparatus and method, and a recording medium. More 
specifically, the present invention relates to an image 
coding apparatus and method, an image decoding apparatus and 
method, and a recording medium which are suitabley for use 
in apparatus for re-encoding video streams and recording and 
reproducing the re-encoded video streams. 

[0003] Digital television broadcasts such as European DVB 
(Digital Video Broadcast), American DTV (Digital Television) 
broadcast, and Japanese BS (Broadcast Satellite) digital 
broadcast use MPEG (Motion Picture Expert Group) 2 transport 
streams. A transport stream consists of continuous 

transport packets, each packet carrying video data or audio 
data, for example. The data length of one transport packet 
is 188 bytes. 

[0004] Unlike analog television broadcasts, digital 
television broadcasts are capable of providing services 
added with multimedia coding data. In these services, data 
such as video data, audio data, character graphics data, and 
still picture data, for example, are associated with each 
other for transmission by the multimedia coding data. For 
the multimedia coding data, a coding method based on XML 
(Extensible Markup Language) is used in the Japanese BS 
digital broadcast, for example. The details of this method 



1 



-ts are disclosed in ARIB STD-B24 Data Coding And 

Transmission Specification for Digital Broadcasting, for 
example . 

[0005] 3rhe dData such as video data, audio data, 

character graphics data, and still picture data are each 
packetized into a transport packet for transmission. 
[0006] FIGS- 1A and IB show an example of synthesizing 
data to be transferred between the sending and receiving 
sides and a multimedia screen. As shown in FIG. 1A, the 
sending side sends to the receiving side video data, 
character graphics data for displaying buttons A through C, 
text data for displaying "XYZABC . . . , " and multimedia coding 
data for relating these data to each other. The sending 
side generally denotes a television broadcast station, for 

example. However, i^fe herein it denotes a television 

broadcast station which includes a recording apparatus (the 
recording side) which receives and records data transmitted 
from broadcast stations, as shown in the example illustrated 
in FIG. 1A including the data which is output from this 
recording apparatus . 

[0007] The multimedia coding data includes data which can 
synthesize on the receiving side video data, character 
graphics data, and text data and display the synthesized 
data. To be more specific, the multimedia coding data 
includes the data associated with the display positions of 
the video, character graphics, and text which are displayed 
by the size-associated data such as the multimedia plane 
(the display area of images on the television receiver, for 
example) size (plane_height and plane_width) and video 
display size ( video_height and video_width) , video data, 
character graphics data, and text data, as shown in FIG. IB. 
[0008] On the basis of the multimedia coding data, the 
receiving side processes the video data, the character 
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graphics data, and the text data to display a resultant 
image, as shown in FIG. IB. 

[0009] Through the screen on which the above-mentioned 
image is displayed, the user can receive services such as 
displaying desired information in the video section by 
clicking button A corresponding to that information and 
obtaining, from the text data displayed in the bottom of the 
screen, the information associated with the matter displayed 
in the video section, for example. 

[0010] If a television program carried by a transport 
stream transmitted from a digital television broadcast is 
recorded without change to a recording medium on the 
received side, the program can be recorded without its 
picture and audio qualities being deteriorated at all. 
However, in order to record as long a television program as 
possible to a recording medium having a limited recording 
capacity by presupposing a certain degree of picture quality 
deterioration, the received video stream must be decoded and 
then encoded again to lower the bit rate of the transport 
stream. 

[0011] For example, the re-encoding of the video stream 
of a television program attached with multimedia coding data 
to lower its bit rate for recording may be implemented by 
sub-sampling the image to change writing blocks. However, 
this approach presents a problem of causing a mismatch in 
the relationship between the video stream resulting from re- 
encoding and the multimedia coding data. The following 
describes an example of this mismatch with reference to 
FIGS. 2A and 2B. 

[0012] In the example shown in FIG. 2A, the sending side 
(the recording side) converts the original video writing 

block to a smaller picture frame at the time of re-encoding. 

Therefore, as shown in FIG. 2B, on the receiving side (the 
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reproducing side) , changes occur in the video display size 
and position, resulting in a display screen which is 
different from the display screen intended by the sending 
side (the display screen to be displayed on the basis of the 
data before being re-encoded) . 
SUMMARY OF THE INVENTION 

[0013] It is therefore an object of the present invention 
to provide an image coding apparatus and method, an image 
decoding apparatus and method, and a recording medium, 
wherein information that there is no mismatch of information 
4rB — caused in the relationship between a video stream after 
re-encoding and other data is generated and recorded on the 
recording side and reproduction on the reproducing side. — 
and wherein — For example, a smaller 4 -s — prevented a — shrunk 
picture frame- *e*e — example is prevented— from occurring on 
the reproducing side by referencing the information 
generated on the recording side. 

[0014] In carrying out the invention and according to a 
first aspect thereof, there is provided a first image coding 
apparatus comprising: inputting means for inputting a 
multiplexed stream containing multimedia coding data; 
separating means for separating a video stream from the 
multiplexed stream input to the separating means from by— the 
inputting means; converting means for performing a 
predetermined converting process on the video stream 
separated by the separating means; generating means for 
generating additional information indicating that a mismatch 
will occur when the converted video stream is displayed on 
the basis of the multimedia coding data; and outputting 
means for outputting the converted video stream, the 
multimedia coding data, and the additional information. 
[0015] The first image coding apparatus may further 
comprise: coding means for coding the additional information 
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generated by the generating means as data separate from the 
multiplexed stream containing the converted video stream. 
[0016] The first image coding apparatus may further 
comprise: coding means for multiplexing the additional 
information generated by the generating means with the 
multiplexed stream containing the converted video stream and 
then coding a multiplexed result. 

[0017] The converting means may convert a video stream 
picture frame parameter. 

[0018] The conversion by the converting means may include 
at least a process of decoding the video stream separated by 
the separating means and a process of encoding the decoded 
video stream. 

[0019] The additional information generated by the 
generating means may contain at least one of an original 
picture frame information and an original screen aspect 
ratio. 

[0020] The additional information generated by the 
generating means may contain an original video format and a 
video format after the conversion. 

[0021] The additional information generated by the 
generating means may contain an original screen aspect ratio 
and a screen aspect ratio after the conversion. 
[0022] The additional information generated by the 
generating means may contain at least one of information 
indicating whether or not a picture frame of the video 
stream has been converted by the converting means, 
information about an original picture frame of the video 
stream separated by the separating means, and an original 
screen aspect ratio . 

[0023] In carrying out the invention and according to a 
second aspect thereof, there is provided a first image 
coding method comprising the steps of: inputting a 
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multiplexed stream containing multimedia coding data; 
separating a video stream from the multiplexed stream input 
in the inputting step; performing a predetermined converting 
process on the video stream separated in the separating 
step; generating additional information indicating that a 
mismatch will occur when the converted video stream is 
displayed on the basis of the multimedia coding data; and 
outputting the converted video stream, the multimedia coding 
data, and the additional information. 

[0024] The first image coding method may further comprise 
the step of: coding the additional information generated in 
the generating step as data separate from the multiplexed 
stream containing the converted video stream. 

[0025] The first image coding method may further 
comprise: a coding step of multiplexing the additional 
information generated in the generating step with the 
multiplexed stream containing the converted video stream and 
then coding a multiplexed result. 

[0026] A video stream picture frame parameter may be 
converted in the converting step. 

[0027] The conversion in the converting step may include 
at least a process of decoding the video stream separated in 
the separating step and a process of encoding the decoded 
video stream. 

[0028] The additional information generated in the 
generating step may contain at least one of original picture 
frame information and an original screen aspect ratio. 
[0029] The additional information generated in the 
generating step may contain an original video format and a 
video format after the conversion. 

[0030] The additional information generated in the 
generating step may contain an original screen aspect ratio 
and a screen aspect ratio after the conversion. 
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[0031] The additional information generated in the 
generating step may contain at least one of information 
indicating whether or not a picture frame of the video 
stream has been converted in the converting step, 
information about an original picture frame of the video 
stream separated in the separating step, and an original 
screen aspect ratio, 

[0032] In carrying out the invention and according to a 
third aspect thereof, there is provided a first method for 
recording a computer-readable program on a first recording 
medium^ recorded — a — computer - readable — program, — the program 
method comprising : the — steps — e#-s — inputting a multiplexed 
stream containing multimedia coding data; separating a video 
stream from the multiplexed stream input — if* — the — inputting 
atop ; performing a predetermined converting process on the 
video stream separated in the from the multiplexed stream to 
form a converted video stream; ocparating — otcp; — generating 
additional information indicating that a mismatch will occur 
when the converted video stream is displayed on the basis of 
the multimedia coding data; and outputting the converted 
video stream, the multimedia coding data, and the additional 
information . 

[0033] In carrying out the invention and according to a 
fourth aspect thereof, there is provided a second recording 
medium having a data recording area, wherein recorded — a 
video stream converted by a predetermined conversion 
process, multimedia coding data, and additional information 
indicating an occurrence of a mismatch at — when displaying 
the converted video stream on the basis of the multimedia 
coding data is recorded . 

[0034] The additional information may be coded and 
recorded as data different from a multiplexed stream 
containing the converted video stream. 
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[0035] The additional information may be coded and 
recorded as multiplexed with a multiplexed stream containing 
the converted video stream. 

[0036] The video stream may be converted in its picture 
frame parameter. 

[0037] The video stream may be decoded and then encoded. 
[0038] The additional information may contain at least 
one of original picture frame information and an original 
screen aspect ratio. 

[0039] The additional information may contain information 
about an original video format and information about a video 
format after the conversion. 

[0040] The additional information may contain information 
about an original screen aspect ratio and information about 
a screen aspect ratio after the conversion. 

[0041] The additional information may contain at least 
one of information indicating whether or not a picture frame 
of the video stream has been converted, information about an 
original picture frame of the video stream, and information 
about an original screen aspect ratio. 

[0042] In carrying out the invention and according to a 
fifth aspect thereof, there is provided a first image 
decoding apparatus comprising: inputting means for inputting 
a multiplexed stream containing multimedia coding data; 
separating means for separating a video stream from the 
multiplexed stream input by the inputting means; decoding 
means for decoding the video stream separated from said 
multiplexed stream by the separating means; and processing 
means for performing a predetermined conversion process on 
the decoded video stream in accordance with additional 
information indicating an occurrence of a mismatch e^fe — when 
displaying the decoded video stream on the basis of the 
multimedia decoding data. 
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[0043] The first image decoding apparatus may further 
comprise: acquiring means for acquiring the additional 
information from data different from — than the multiplexed 
stream. 

[0044] The first image decoding apparatus may further 
comprise: acquiring means for acquiring the additional 
information from a multiplexed stream with — which containing 
the additional information is multiplexed . 

[0045] The processing means may convert a picture frame 
parameter of the video stream. 

[0046] The conversion by the processing means may include 
at least a process of decoding the video stream separated by 
the separating means from said multiplexed stream and a 
process of encoding the decoded video stream. 

[0047] The additional information may contain at least 

one of information about an original picture frame and 

information about an original screen aspect ratio. 

[0048] The additional information may contain an original 

video format and information about a video format after the 

conversion. 

[0049] The additional information may contain information 
about an original screen aspect ratio and information about 
a screen aspect ratio after the conversion. 

[0050] The additional information may contain at least 
one of information indicating whether or not a picture frame 
of the video stream has been converted by the converting 
means, information about an original picture frame of the 
video stream separated by the separating means, and 
information about an original screen aspect ratio. 
[0051] In carrying out the invention and according to a 
sixth aspect thereof, there is provided a first image 
decoding method comprising the steps of: inputting a 
multiplexed stream containing multimedia coding data; 
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separating a video stream from the multiplexed stream input 
in the inputting step; decoding the video stream separated 
in the separating step; and performing a predetermined 
conversion process on the decoded video stream in accordance 
with additional information indicating an occurrence of a 
mismatch e^ -when displaying the decoded video stream on the 
basis of the multimedia decoding data. 

[0052] The first image decoding method may further 
comprise the step of: acquiring the additional information 
from data different from than the multiplexed stream. 
[0053] The first image decoding method may further 
comprise the step of: acquiring the additional information 

from a multiplexed stream with which containing the 

additional information is multiplexed . 

[0054] A picture frame parameter of the video stream may 
be converted in the processing step. 

[0055] The conversion in the processing step may include 
at least a process of decoding the video stream separated 
from the multiplexed stream in the separating step and a 
process of encoding the decoded video stream. 

[0056] The additional information may contain at least 
one of information about an original picture frame and 
information about an original screen aspect ratio. 
[0057] The additional information may contain an original 
video format and information about a video format after the 
conversion . 

[0058] The additional information may contain information 
about an original screen aspect ratio and information about 
a screen aspect ratio after the conversion. 

[0059] The additional information may contain at least 
one of information indicating whether or not a picture frame 
of the video stream has been converted in the converting 
step, information about an original picture frame of the 
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video stream separated in the separating step, and 
information about an original screen aspect ratio. 
[0060] In carrying out the invention and according to a 
seventh aspect thereof, there is provided a third — second 
method for recording medium — recorded — a computer-readable 
progra m on a third recording medium , the program — method 
comprising the steps of: inputting a multiplexed stream 
containing multimedia coding data; separating a video stream 
from the multiplexed stream input in the inputting step; 
decoding the video stream separated from the multiplexed 
stream in the separating step; and performing a 
predetermined conversion process on the decoded video stream 
in accordance with additional information indicating 
occurrence of a mismatch efe — when displaying the decoded 
video stream on the basis of the multimedia decoding data. 
[0061] In carrying out the invention and according to an 
eighth aspect thereof, there is provided a second image 
coding apparatus comprising: inputting means for inputting a 
multiplexed stream; separating means for separating a video 
stream from the multiplexed stream input by the inputting 
means; determining means for determining whether e^e — aet- 
multimedia coding data is contained in the multiplexed 
stream input by the inputting means; generating means 7 — if it 
i-s — determined — by — the — determining — means — that — t-he — multimedia 
coding — data — i-s — contained — i-e — the — multiplexed — stream, for 
generating coding control information based on the presence 
of said multimedia coding data for instructing f*efe — fee — change 
that a display format of the video stream separated by the 
separating means is not changed ; converting means for 
performing a predetermined conversion process on the video 
stream separated by the separating means on the basis of the 
coding control information generated by the generating 
means; and multiplexing means for generating a multiplexed 



11 



stream which that contains the video stream converted by the 
converting means. 

[0062] The generating means may provide an instruct ion 
not to change any of a picture frame, a video format, and an 
aspect ratio. 

[0063] In carrying out the invention and according to a 
ninth aspect thereof, there is provided a second image 
coding method comprising the steps of: inputting a 
multiplexed stream; separating a video stream from the 
multiplexed stream input in the inputting step; determining 
whether e^e — n^fe — multimedia coding data is contained in the 
multiplexed stream input in the inputting step; generating 
coding control information based on the presence of said 
multimedia coding data in said multiplexed stream for 
instructing f*e£ — fce — change that a display format of the video 
stream separated in the separating step is not changed; 7 — i# 
i£ — i-s — determined in the determining — step that — the multimedia 

coding data i-s contained if* the multiplexed stream; 

performing a predetermined conversion process on the video 
stream separated in the separating step on the basis of the 
coding control information generated in the generating step; 
and generating a multiplexed stream which that contains the 
video stream converted in the converting step. 

[0064] Coding control information for providing an 
instruct ioni rftg^ not to change any of a picture frame, a video 
format, and an aspect ratio may be generated in the 
generating step. 

[0065] In carrying out the invention and according to a 
tenth aspect thereof, there is provided a third method for 
fourth recording medium recorded a computer-readable program 
on a fourth recording medium , the program m ethod comprising 
the steps of: inputting a multiplexed stream; separating a 
video stream from the multiplexed stream input in the 
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inputting step; determining whether or not m ultimedia coding 
data is contained in the multiplexed stream input in the 
inputting step; generating coding control information based 
on the presence of the multimedia coding in the multiplexed 
stream for instructing net — fee — change — that a display format 
of the video stream separated in the separating step is not 
changed 7 — ±£ — it — ±b — determined — if* — the — determining — step — that 
trhe — multimedia — coding — data — ts — contained — if* — the — multiplexed 
otrcam ; performing a predetermined conversion process on the 
video stream separated in the separating step on the basis 
of the coding control information generated in the 
generating step; and generating a multiplexed stream which 
that contains the video stream converted in the converting 
step . 

[0066] In carrying out the invention and according to an 
eleventh aspect thereof, there is provided a fifth recording 
medium comprising a data recording area, wherein recorded 
coding control information is recorded for instructing n-et- 
te — change — that a display format is not changed for e £ — a 
video stream and a multiplexed stream containing a video 
stream on which a predetermined conversion process has been 
performed on the basis of the coding control information . 
[0067] As described and according to the first image 
coding apparatus and method and the program stored in the 
first recording medium, a video steam is separated from a 
multiplexed stream containing multimedia coding data, and a 
predetermined conversion process is performed on the 
separated video stream, and additional information 
indicating that a mismatch will occur when displaying the 
converted video stream on the basis of the multimedia coding 
data . 

[0068] The second recording medium stores a video stream 
converted by a predetermined conversion process, the 
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multimedia coding data, and the additional information 
indicating that a mismatch will occur when displaying the 
converted video stream on the basis of the above-mentioned 
multimedia coding data. 

[0069] As described and according to the image decoding 
apparatus and method and the program stored in the third 
recording medium, when a video stream is separated from an 
input multiplexed stream, the separated video stream is 
decoded, and the decoded video stream is displayed on the 
basis of multimedia coding data, a mismatch occurs. On the 
basis of the additional information about this mismatch 
occurrence, a predetermined conversion process is performed 
on the decoded video stream. This novel configuration 
prevents the mismatch from occurring between the video 
stream and the multimedia coding data. 

[0070] As described and according to the second image 
coding apparatus and method and the program stored in the 
fourth recording medium, a video stream is separated from an 
input multiplexed stream, the input multiplexed stream is 
checked whether or not multimedia coding data is contained 
and, if the multimedia coding data is contained, coding 
control information for giving an instruction not to change 
the display format of the separated video stream is 
generated, and a predetermined conversion process is 
performed on the separated video stream on the basis of the 
generated coding control information. 

[0071] The fifth recording medium also stores the above- 
mentioned coding control information giving instruction not 
to change the display format of a video stream and a 
multiplexed stream containing the video stream on which a 
predetermined conversion process has been performed on the 
basis of the coding control information. 
BRIEF DESCRIPTION OF THE DRAWINGS 



14 



[0072] These and other objects of the invention will be 
seen by reference to the description, taken in connection 
with the accompanying drawings^ in which: 

[0073] FIGS. 1A and IB are schematic diagrams 
illustrating a display screen to be shown on the basis of 
multimedia coding information; 

[0074] FIGS. 2A and 2B are schematic diagrams 
illustrating a mismatch which takes place when a video 
stream is re-encoded; 

[0075] FIG. 3 is a block diagram illustrating a recording 
apparatus practiced as one embodiment of the present 
invention; 

[0076] FIGS. 4A and 4B illustrate the operation of a 
multiplexer shown in FIG. 3; 

[0077] FIGS. 5A, 5B and 5C illustrate the processing by 
an arrival timestamp adding block; 

[0078] FIG. 6 illustrates multimedia display sub- 
information; 

[0079] FIG. 7 illustrates an example of Programlnf o ( ) 
syntax; 

[0080] FIG. 8 illustrates an example of 

StreamCodinglnf o ( ) syntax; 

[0081] FIG. 9 illustrates the meaning of stream_coding 
type; 

[0082] FIG. 10 illustrates the meaning of video_f ormat ; 
[0083] FIG. 11 illustrates the meaning of frame_rate; 
[0084] FIG. 12 illustrates the meaning of 

display_aspect_ratio; 

[0085] FIG. 13 is a flowchart describing the processing 
of coding AV stream and multimedia display sub-information; 

[0086] FIG. 14 is a flowchart describing the coding 
processing to be executed for restricting the re-encoding of 
a multiplexed stream video including multimedia coding data; 
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[0087] FIG- 15 illustrates an example of an input 
transport stream; 

[0088] FIG. 16 illustrates an example of a transport 
stream after the re-encoding of the video stream shown in 
FIG. 15; 

[0089] FIG. 17 is a flowchart describing a recording rate 
control process by a recording apparatus shown in FIG. 3; 
[0090] FIG. 18 is a flowchart describing another 
recording rate control process by the recording apparatus 
shown in FIG. 3; 

[0091] FIG. 19 illustrates another example of a transport 
stream resulting from the re-encoding of the video stream; 
[0092] FIG. 20 illustrates another example of the input 
transport stream; 

[0093] FIG. 21 is a block diagram illustrating a 
configuration of a reproducing apparatus practiced as one 
embodiment of the present invention; 

[0094] FIGS. 22A and 22B illustrate a display screen to 
be shown when multimedia display sub-information is added; 
[0095] FIG. 23 is a block diagram illustrating another 
configuration of the recording apparatus practiced as one 
embodiment of the present invention; 

[0096] FIG. 24 is a flowchart describing the processing 
of reproducing an AV stream which uses multimedia display 
sub- in format ion; 

[0097] FIG. 25 is a block diagram illustrating another 
configuration of the reproducing apparatus practiced as one 
embodiment of the present invention; and 
[0098] FIG. 26 illustrates recording media. 
DETAILED DESCRIPTION 

[0099] This invention will be described in further detail 
by way of example with reference to the accompanying 
drawings. Now, referring to FIG. 3, there is shown a block 
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diagram illustrating an exemplary configuration of a 
recording apparatus 1 practiced as one embodiment of the 
invention. A transport stream received at an antenna, not 
shown, is input in a selector 10- A program number (a 
channel number) specified by the user is also input from a 
terminal 11 to the selector 10- Referring to the received 
program number, the selector 10 extracts the specified 
program from the received transport stream and outputs a 
partial transport stream. The partial transport stream is 
input in a demultiplexer 12 and an analyzing block 13. 
[0100] The partial transport stream input in the 
demultiplexer 12 is separated into a video stream and other 
streams (audio, still picture, character graphics, and 
multimedia coding data for example) . The video stream thus 
obtained is output to a decoder 14. The other streams are 
output to a multiplexer 16. In addition to the transport 
packets other than video, the demultiplexer 12 outputs the 
output timing information in the input transport stream of 
these transport packets to the multiplexer 16. 

[0101] The decoder 14 applies a predetermined decoding 
scheme, for example, MPEG2 to the input video stream and 
outputs the decoded video data to an encoder 15. Also, the 
decoder 14 outputs the stream information about the video 
stream obtained at decoding to a coding controller 18. 
[0102] On the other hand, the analyzing block 13 analyzes 
the input transport stream to obtain the stream information 
about the non-video streams, for example, a bit rate^ and 
outputs it to the coding controller 18. The stream 

information about the non-video streams output from the 
analyzing block 13, the video stream information output from 
decoder 14, and a stream recording bit rate output from a 
terminal 19 are input in the coding controller 18. From 
these data, the coding controller 18 sets the video data 
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coding conditions (coding control information) to be 
executed by the encoder 15 and outputs these coding 
conditions to the encoder 15 and a coding block 20. 
[0103] The coding controller 18 uses, as a bit rate to be 
allocated to the video data encoding, a value obtained by 
subtracting a total value (the data input from the analyzing 
block 13) of the bit rates of the non-video streams from a 
stream recording bit rate (the data input, via the terminal 
19, from a controller, not shown, for controlling the 
operation of the recording apparatus 1, for example) - The 
coding controller 18 sets coding control information such as 
bit rate and picture frame such that an optimum picture 
quality can be achieved with the bit rate thus obtained and 
outputs this coding control information to the encoder 15 
and the coding block 20. The details of the coding control 
information will be described later with reference to FIGS. 
15 through 20. 

[0104] When a stream is recorded to a recording medium 
with a fixed rate, this stream recording bit rate becomes 
the fixed rate; if a stream is recorded with a variable bit 
rate, this stream recording bit rate is a mean bit rate per 
predetermined time. However, the maximum value of the 
variable bit rate in this case needs to be lower than the 
maximum recording bit rate ensured by the recording medium 
concerned . 

[0105] The encoder 15 encodes (on the basis of MPEG2^ for 
example) the video data output from the decoder 14 on the 
basis of the coding control information output from the 
coding controller 18 and outputs the resultant video data to 
the multiplexer 16. The video stream from the encoder 15, 
the transport stream packets other than video from the 
demultiplexer 12, and the information about the occurrence 
timing of the transport stream packets other than video are 
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input in the multiplexer 16. On the basis of the input 
occurrence timing information, the multiplexer 16 
multiplexes the video stream with the transport stream 
packets^ other than video^_ and outputs the result to the 
arrival timestamp adding block 17 as a transport stream. 
[0106] FIGS. 4A and 4B schematically illustrate the 
above-mentioned processing to be executed by the multiplexer 
16. FIG. 4A shows the timing of the input transport stream 
packets. In these figures, the cross-hatched portions 
indicate the video packets while the white portions indicate 
the stream packets other than video. As shown in FIG. 4A, 
the input transport stream packets are continuous; however, 
the data volume of the video data is reduced by the re- 
encoding of video data by the encoder 15. Consequently, the 
number of video packets is reduced. 

[0107] As shown in FIG. 4B, the multiplexer 16 does not 
change the timing of the stream packets other than video but 
causes only the timing of the video packets to be different 
from the original state (shown in FIG. 4A) . 

[0108] As shown in FIGS. 5A, 5B and 5C, the arrival 
timestamp adding block 17 adds a header (TP__extra_header ) 
including an arrival timestamp to each of the packets (FIG. 
5A) of the input transport stream to generate a source 
packet (FIG. 5B) , arranges the generated source packets 
continuously (FIG. 5C) , and outputs them to a writing block 
21. The arrival timestamp is information indicative of the 
timing with which the transport stream packets occur in a 
transport stream. The writing block 21 puts takes the input 
source packet stream consisting of continuous source packets 
and records the file to a recording medium 22. It should be 
noted that the recording medium 22 may be any type of 
recording medium. 

[0109] The information output from the coding block 20 is 
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also input in the writing block 21. On the basis of the 
video coding information from the coding controller 18, the 
coding block 20 generates multimedia display sub-information 
and outputs the same to the writing block 21. The 
multimedia display sub-information to be output to the 
writing block 21 is information for keeping the video 
display position and display size unchanged on multimedia 
plane from those of the image (the image which would be 
displayed without re-encoding) intended by the sending side 
even if the picture frame size has changed by transcoding 
(decoding by the decoder 14 and then encoding by the encoder 
15) a video stream. This information io also is used at the 
time of reproduction in combination with multimedia coding 
data . 

[0110] The following describes the multimedia display 
sub-information more specifically. As shown in FIG. 6, the 
multimedia display sub-information consists of three flags 
of a mismatch flag (mismatch_MMinf o_f lag) , a re-encoded flag 
(Re_encoded_f lag) , and a frame size change flag 
(changed_f rame_size_f lag) , data associated with two sizes 
indicative of an original horizontal size 

(original_horizontal_size) and an original vertical size 
(original_vertical__size) , and an original screen aspect 
ratio (original_display_aspect_ratio) . 

[0111] The mismatch flag indicates whether there exists a 
mismatch in the relationship between video and multimedia 
coding data. The re-encoded flag indicates whether the 
video has been re-encoded at the time of recording. The 
frame size change flag indicates whether the picture frame 
of video has been changed by re-encoding_^ for example. The 
original horizontal size indicates the horizontal size of a 
picture frame before re-encoding. The original vertical 
size indicates the vertical size of a picture frame before 
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re-encoding. The original screen aspect ratio indicates the 
aspect ratio of a frame screen before re-encoding. 
[0112] It should be noted that the above-mentioned 
multimedia display sub-information is illustrative only. 
Therefore, information other than that shown in FIG. 6 may 
be included in^_ or part of the information shown in FIG. 6 
may be excluded from^ the multimedia display sub- 
information. 

[0113] The following describes another example of the 
multimedia display sub-information. In the following 

example, the multimedia display sub-information is stored in 
a Programlnf o ( ) syntax shown in FIG. 7. The following 
describes the fields associated with the present invention 
in the Programlnf o ( ) syntax. 

[0114] "length" indicates the number of bytes between the 
byte just after the length field and the last byte of 
Programlnf o ( ) inclusive . 

[0115] "num_of_program_sequences" indicates the number of 
program sequences in the an AV stream file. A source packet 
sequence with which the program contents specified by this 
format in the AV stream file are constant is referred to as 
a program sequence. 

[0116] "SPMN program sequences start" indicates an 

address at which the program sequence starts in the AV 
stream file. "SPMN program sequences start" is of a size in 
unit of source packet number and counted from the initial 
value 0 starting with the first packet of the AV stream 
file. 

[0117] "program_map_PID" is value of the PID of a 
transport packet having PMT (Program Map Table) applicable 
to that program sequence. 

[0118] "num_of__streams_in_ps" indicates the number of 

elementary streams defined in that program sequence. 
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[0119] "stream_PID" indicates the value of the PID for 
the elementary stream defined in the PMT which is referenced 
by the program map PID of that program sequence. 
[0120] "StreamCodinglnf o ( ) " indicates the information 
about the elementary stream indicated by the above-mentioned 
stream PID. 

[0121] FIG . 8 shows the syntax of StreamCodinglnf o () . 
"length" indicates the number of bytes between the byte j ust 
after this length field and the last byte of 
StreamCodinglnf o ( ) inclusive . 

[0122] "stream_coding_type" indicates the coding type of 
the elementary stream indicated by the stream PID for this 
StreamCodinglnf o () . The meanings of the individual types 
are shown in FIG. 9 . 

[0123] If the value of stream coding type is 0x02, it 
indicates that the elementary stream indicated by the stream 
PID is a video stream. 

[0124] If the value of stream coding type is OxOA, OxOB, 
or OxOD, it indicates that the elementary stream indicated 
by the stream PID is multimedia coding data. 

[0125] If the value of stream coding type is 0x06, it 
indicates that the elementary stream indicated by the stream 
PID is subtitles or teletext. 

[0126] "video_f ormat" indicates the video format of a 
video stream indicated by the stream PID for this 
StreamCodinglnf o () . The meanings of the individual video 
formats are shown in FIG. 10 . 

[0127] In FIG. 10, 480i indicates video display of NTSC 

standard TV (interlace frame of 720 pixels x 480 lines) . 
576i indicates video display of PAL standard TV (interlace 
frame of 720 pixels x 576 lines) - 480p indicates video 
display of progressive frame of 720 pixels x 480 lines. 
1080i indicates video display of interlace frame of 1920 
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pixels x 1080 lines, 720p indicates video display of 
progressive frame of 1230 pixels x 720 lines. 

[0128] "f rame_rate" indicates the frame rate of a video 
stream indicated by the stream PID for this 
StreamCodinglnf o () . The meanings of the individual frame 
rates are shown in FIG. 11. 

[0129] "display_aspect_ratio" indicates the display 
aspect ratio of a video stream indicated by the stream PID 
for this StreamCodinglndo ( ) . The meaning of the individual 
display aspect ratios are shown in FIG . 12. 

[0130] "original_video_f ormat_f lag" indicates whether 
there exists^ original video format and original display 
aspect ratio in this StreamCodinglnf o () . 

[0131] "original_video_f ormat " indicates a video format 
before a video stream indicated by the stream PID for this 
StreamCodinglnf o ( ) is coded. The meanings of the individual 
original video formats are the same as shown in FIG. 10. 
[0132] "original_display_aspect_ratio" is the display 
aspect ratio before a video stream indicated by the stream 
PID for this StreamCodinglnf o ( ) is coded. The meanings of 
the individual aspect ratios are the same as shown in FIG. 
12. 

[0133] It is assumed that, in transcoding a transport 
stream with a multimedia data stream (BML stream or 
subtitles) multiplexed along with a video stream, the re- 
encoding of the video stream changes its video format (for 
example, from 1080i to 480i) , while the multimedia data 
stream retains its original stream contents. In this case, 
a mismatch in information may occur between a new video 
stream and the multimedia data stream. For example, 

although the parameters associated with the display of the 
multimedia data stream are determined on the supposition of 
the video format of the original video stream, the video 
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format may be changed by the re-encoding of the video 
stream. 

[0134] The video format of the original video stream is 
indicated by the video format and the display aspect ratio. 
The video format of the re-encoded video stream is indicated 
by the original video format and the original display aspect 
ratio. 

[0135] If there — exists — a mismatch exists between the 
values of the video format and the original video format 
and/or between the display aspect ratio and the original 
display aspect ratio, it indicates that a video format 
change has been caused by the video re-encoding at the time 
of recording. 

[0136] If the stream PID in which the stream coding type 

indicates multimedia coding data and subtitles 4r3 are 

included in Programlnf o ( ) , it indicates that the multimedia 
data is multiplexed in an AV stream file (a transport 
stream) . 

[0137] If Programlnf o ( ) indicates that a video format 
change has been caused by the re-encoding of video at the 
time of recording and multimedia data is multiplexed in the 
AV stream file, then it is determined that there — exists — a 
mismatch exists in display between the video stream (re- 
encoded) and the multimedia data (the original multimedia 
data) in the AV stream file. 

[0138] In such a case, the information about the original 
video stream, namely the original video format and the 
original display aspect ratio, becomes effective. The 
reproducing apparatus generates a display screen from the 
above-mentioned new video stream and multimedia data stream 
as follows. 

[0139] The video stream is up-sampled to a video format 
indicated by the original video format and the original 
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display aspect ratio. 

[0140] The up-sampled image and the multimedia data 
stream are synthesized to form a correct display screen. 
[0141] The multimedia display sub-information generated 
by the coding block 20 is recorded by the writing block 21 
to the recording medium 22 but stored as a file which is 
different from the source packet stream file output from the 
arrival timestamp adding block 17. If the multimedia 
display sub-information is recorded by the writing block 21 
to the recording medium 22 as a file different from the 
source packet stream file, the filed multimedia display sub- 
information is output from the coding block 20. 
[0142] FIG. 13 is a flowchart describing the processing 
of coding an AV stream and multimedia display sub- 
information . 

[0143] In step 50, a multiplexed stream including 
multimedia coding data is input in the recording apparatus 
1. 

[0144] In step 51, the demultiplexer 12 separates the 
video stream from the multiplexed stream. 

[0145] In step 52, the encoder 15 re-encodes the video 
stream decoded by the decoder 14 . 

[0146] In step 53, the multiplexer 16 multiplexes the 
above-mentioned video stream and multimedia coding data to 
generate a multiplexed stream. 

[0147] In step 54, the coding block 20 generates 

multimedia display sub-information . 

[0148] In the above description, the coding controller 18 
generates the coding control information including bit rate 
and picture frame on the basis of the input data. The 
coding controller 18 may generate the following information 
as alternative coding control information. Namely, if the 
input transport stream is found to_includeif*^ multimedia 
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coding data by the analyzing block 13, then the coding 
controller 18 may generate coding control information— when 
encoding is executed by the encoder 15— for instructing the 
encoder 15 to execute the re-encoding with a picture frame 
(the picture frame before re-encoding) of the same size as 
that of the picture frame of the original video^ and output 
the generated coding control information to the encoder 15. 
[0149] When the above-mentioned method is used, the 
encoder 15 re-encodes the video data supplied from the 
decoder 14 with the same value as that of the picture frame 
of the original video stream on the basis of the input 
coding control information. If such coding control 

information is generated and the re-encoding is executed on 
the basis of the coding control information, no picture 
frame change is caused by the re-encoding, thereby 
preventing a mismatch from occurring in the relationship 
between the video stream obtained by re-encoding and the 
multimedia coding data. 

[0150] Still alternatively, the following information may 
be generated as the coding control information generated by 
the coding controller 18. Namely, if the input transport 
stream is found to include ing multimedia coding data by the 
analyzing block 13, then the coding controller 18 may 
generate coding control information— when encoding is 
executed by the encoder 15— for instructing the encoder 15 
to execute the re-encoding under the same conditions as the 
video format (shown in FIG. 10) and screen aspect ratio 

(shown in FIG. 12) of the original video_^ and output the 
coding control information to the encoder 15. 

[0151] When the above-mentioned method is used, the 
encoder 15 re-encodes the video supplied from the decoder 14 
under the same conditions as the video format (shown in FIG. 
10) and screen aspect ratio (shown in FIG. 12) of the 
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original video on the basis of the input coding control 
information. If such coding control information is 

generated and the re-encoding is executed on the basis of 
the coding control information, no video format and no 
screen aspect ratio change is caused by the re-encoding, 
thereby preventing a mismatch from occurring in the 
relationship between the video stream obtained by re- 
encoding and the multimedia coding data. 

[0152] FIG. 14 is a flowchart describing the coding for 
restricting the re-encoding of the video of a multiplexed 
stream including multimedia coding data. 

[0153] In step 70, a multiplexed stream is input in the 
recording apparatus 1. 

[0154] In step 71, the demultiplexer 12 separates the 
video stream from the multiplexed stream. 

[0155] In step 72, the analyzing block 13 checks if the 
multimedia coding data is included in the video stream. If 
the multimedia coding data is included, the analyzing block 
13 sends the coding control information to the encoder 15 
instructing the same to re-encode the video stream without 
changing the display format. On the basis of the supplied 
control information, the encoder 15 re-encodes the video 
stream. 

[0156] In step 73, the multiplexer 16 generates a 
multiplexed stream including the above-mentioned video 
stream. 

[0157] With reference to FIGS. 15 through 20, ^ the 
following describes one example of control to be executed on 
the basis of the coding control information — with — reference 
to FIGS. 15 through 20 . 

[0158] It is assumed here that a transport stream to be 
input in to the selector 10 has¥e a constant bit rate Ri as 
shown in FIG. 15^_ for example. The video stream and the 
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non-video streams are coded by variable bit rates. In the 
example shown in FIG . 15 , in unit time (for example, GOP) A, 
the bit rate of the video stream is R V a and the bit rate of 
non-video streams is R 0 a- In unit time B, the bit rate of 
the video stream is R VB and the bit rate of non-video streams 
is Rob- In unit time C, the bit rate of the video stream is 
Rvc and the bit rate of non-video streams is Roc- 
[0159] If the transport stream as shown in FIG . 15 is re- 
encoded to output the transport stream having fixed bit rate 
S (S < Ri) as shown in FIG. 16 from the multiplexer 16, the 
coding controller 18 executes the processing described by 
the flowchart shown in FIG. 17. 

[0160] First, in step SI, the coding controller 18 sets 
•fee — S — the bit rate to S (recording rate) of a transport 
stream to be output from the multiplexer 16 on the basis of 
a control signal input from a controller, not shown, via the 
terminal 19. Next, in step S2, the coding controller 18 
determines non-video streams to be recorded and computes a 
maximum total value D of the bit rates of the determined 
streams . 

[0161] The maximum value D is determined from the stream 
specification of the input transport stream. For example, 
if two audio streams are to be recorded in addition to the 
video stream, the maximum value D is 384 x 2 Kbps since the 
maximum value of the bit rate of one audio stream is 384 
Kbps according to the Japanese digital BS broadcast stream 
specification . 

[0162] In step S3, the coding controller 18 uses value C 
obtained by subtracting the maximum value D computed in step 
S2 from the recording bit rates- set in step SI (C= S - D) , 
as a bit rate to be allocated to the re-encoding of the 
video data. In step S4, the coding controller 18 analyzes 
the coding information such as the video stream bit rate and 
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picture frame from the video stream information output from 
the decoder 14. 

[0163] In step S5, the coding controller 18 determines, 
on the basis of the value C computed in step S3 and the 
video stream coding information analyzed in step S4, a video 
coding parameter (video coding control information) such 
that an optimum picture quality is achieved. 

[0164] For example, in the example shown in FIG. 16, 
value S is 1/2 of value R z . In the present example, the bit 
rate of steams other than video is the maximum value D, 
which is used without change as the bit rate of non-video 
steams in a multiplexed stream after re-encoding. 
[0165] Then, video coding parameters are determined such 
that an optimum picture quality can be achieved within the 
range of (S - D) . If the picture frame is controlled, the 
horizontal direction of a picture frame of 720 x 480 pixels^ 
for example^ is sampled by 1/2 into 360 x 480 pixels. The 
determined coding parameters (bit rate and picture angle) 
are supplied to the encoder 15 as video coding control 
information . 

[0166] In step S6, on the basis of the video coding 
control information supplied from the coding controller 18, 
the encoder 15 re-encodes the video data of unit time (in 
this example, unit time A) to be processed now. In the 
example shown in FIG. 16, the actual bit rate R 0 a is smaller 
than the maximum value D in unit time A; however, since the 
maximum value D is fixed, the video allocated bit rate 
becomes (S - D) . A wasted portion Rsa which cannot be used 
for video coding occurs because the maximum value D is 
fixed^— _J^The wasted portion being — is filled with stuffing 
bits . 

[0167] In step S7, the coding controller 18 determines 
whether there remains any stream to be re-encoded. If there 
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still — remains — any strea ms remain to be re-encoded, the 
procedure returns to step S4 to repeat the above-mentioned 
processes . 

[0168] If, in step SI, thorc — remains — no more streams 
remain to be re-encoded, this processing comes to an end. 
[0169] Thus, in the example shown in FIG. 16, in unit 
time B, the bit rate of non-video streams also is D also in 
unit time B and the video stream allocated bit rate is S - D 
because it is fixed. Stuffing bits are inserted in value R sb 
(R S b_= S - (S - D) - Rob = D - Rob) - 

[0170] In unit time C, too, the bit rate of non-video 
streams is D and the video stream allocated bit rate is S - 
D. It should be noted that, in unit time C, 

[0171] D = Roc, so that there — exists — no stuffing bits 
exist . 

[0172] Thus, in the example shown in FIG. 16, the video 
stream is coded with a fixed bit rate. 

[0173] FIG. 18 is a flowchart describing a processing 
example in which the video re-encoding allocated bit rate is 
variable. First, in step S21, the coding controller 18 sets 
recording rate S on the basis of the information supplied 
via the terminal 19. Next, in step S22, the coding 
controller 18 analyzes the coding information of the video 
stream on the basis of the video stream information supplied 
from the decoder 14. The processes of steps S21 and S22 are 
the same as those of steps SI and S4 of FIG. 17. 
[0174] In step S23, the coding controller 18 computes, 
from the output of the analyzing block 13, the total bit 
rate B in each unit time of non-video streams. 
[0175] In step S24, the coding controller 18 uses, as the 

video re-encoding allocated bit rate, value C (C = S - B) 

obtained by subtracting value B obtained in step S23 from 
value S obtained in SI. 
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[0176] In step S25, the coding controller 18 determines, 
on the basis of value C obtained in step S24 and a result of 
analysis of the video stream coding information obtained in 
step S22, video coding parameters such that an optimum 
picture quality is obtained- The determined coding 

parameters are output to the encoder 15. 

[0177] In step S26, the encoder 15 re-encodes the video 
data of the current unit time on the basis of the coding 
parameters determined in step S25. Consequently, as shown 
in FIG • 19j_ for example, after allocation of R oa (=Roa) as the 
bit rate in unit time of non-video streams, the bit rate of 
the video stream is set to bit rate R va specified by (S - 

Roa) • 

[0178] In step S27, the coding controller 18 determines 
whether thorc — remains — any stream s remain to be processed. 
If there — still — remains — any stream s remain to be processed, 
the procedure returns to step S22 to repeat the above- 
mentioned processes. If there — remains — no more streams 
remain to be processed, this processing comes to an end. 
[0179] Thus, in unit time B, after allocation of bit rate 
Rob (= S - R 0 b) of non-video streams, the remaining R V b (= S - 
R ob ) is the bit rate of the video stream. In unit time C, 
the bit rate of the video stream is set to R vc (= S - Roc)_r_ 
except for bit rate Roc of non-video streams. 

[0180] Thus, in the present processing example, the bit 
rate of the video stream is variable and^ therefore^ no 
stuffing bit is needed or the number of stuffing bits can be 
reduced, thereby coding the video stream more efficiently. 
[0181] In the above, the input transport stream has a 
fixed bit rate. The present invention is also is applicable 
to an example in which the bit rate of the input transport 
stream is variable as shown in FIG. 20. 

[0182] Consequently, a transport stream of longer content 
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can be recorded to the recording medium 22 at a lower bit 
rate as required. 

[0183] In addition, the above-mentioned novel 

constitution — embodiment prevents the qualities of audio 
data, still picture and character graphics data, multimedia 
coding data, and other non-video data from being 
conspicuously deteriorated. The-se non-video data is 

basically smaller in data volume than video data, so that 
reducing theia? bit rate of the non-video data in the same 
ratio as the bit rate of video data makes the effects on 
theee non-video data relatively greater than those on video 
data. The novel constitution embodiment can prevents these 
effects from being caused. 

[0184] The following describes the reproduction of a 
source packet stream file recorded on the recording medium 
22. Referring to FIG. 21, there is shown a block diagram 
illustrating the configuration of a reproducing apparatus 
practiced as one embodiment of the invention. A source 
packet stream file recorded on the recording medium 22 is 
read by a reading block 31. The reading block 31 also reads 
multimedia display sub-information recorded on the recording 
medium 22 as a file separate from the source packet stream 
file. 

[0185] The source packet stream read by the reading block 
31 is output to a arrival timestamp separating block 32 and 
the multimedia display sub-information is output to a 
synthesizing block 36. The arrival timestamp separating 
block 32 incorporates a reference clock. The arrival time 
stamp separating block 32 compares the reference clock with 
the value of the arrival timestamp added to the source 
packet of the input source packet stream and, when a match 
is found, removes the arrival timestamp from the source 
packet having the matching arrival timestamp, outputting the 
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resultant packet to a demultiplexer 33 as a transport stream 
packet . 

[0186] The demultiplexer 33 separates the input transport 
stream into a video/audio stream and data streams such as 
multimedia coding data, character graphics, text, and still 
picture. Of these separated data, the video/audio stream is 
output to an AV decoder 34, the multimedia coding data is 
output to the synthesizing block 36, and the data stream 
such as character graphics, text, and still picture is 
output to a character graphics/still picture decoder 35. 
[0187] The AV decoder 34 separates the input video/audio 
stream into video data and audio data, decodes each data, 
and outputs the decoded audio data to an audio reproducing 
device, not shown, and the decoded video data to the 
synthesizing block 36 . The character graphics /still picture 
decoder 35 decodes the input data stream^ such as character 
graphics, text, and still picture^ and outputs the decoded 
character graphics data, text data, and still picture data 
to the synthesizing block 36. 

[0188] In the synthesizing block 36, the video data from 
the AV decoder 34, the multimedia coding data from the 
demultiplexer 33, the data from the character graphics/still 
picture decoder 35, and the multimedia display sub- 
information from the reading block 31 are input. Checking 
the mismatch flag (FIG. 6) of the input multimedia display 
sub-information, the synthesizing block 36 determines 
whether there — exits — a mismatch exists in the relationship 
between the input video signal and the multimedia coding 
data . 

[0189] If there — cxioto — a mismatch exists between the 
value of video format and the value of original video format 
shown in FIG. 8 and/or a mismatch exists between the value 
of display aspect ratio and the original display aspect 
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ratio, the synthesizing block 36 determines that a video 
format change has been caused by the video re-encoding at 
the time of recording, detecting a mismatch in the 
relationship between the input video signal and the 
multimedia encoding data. If there — exists — no mismatch 
exists between the value of video format and the value of 
original video format and no mismatch exists between the 
value of display aspect radio and the value of original 
display aspect ratio, the synthesizing block 36 determines 
that there — exists — no mismatch exists in the relationship 
between the input video signal and the multimedia coding 
data . 

[0190] If a mismatch is found in the relationship between 
the input video signal and the multimedia coding data, the 
synthesizing block 36 further references the original 
horizontal size and vertical size of the multimedia display 
sub-information or references the original video format and 
the original display aspect ratio. Then, the synthesizing 
block 36 scale-converts the input video signal so that it 
can be displayed in a frame of the referenced size. On the 
basis of the multimedia coding data, the synthesizing block 
36 outputs the video signal with the scale-converted video 
signal and the data^ such as character graphics synthesized 
on a multimedia plane_^ to a television receiver, not shown, 
which servers as a display device. 

[0191] On the other hand, if no mismatch is found in the 
relationship between the input video signal and the 
multimedia coding data, the synthesizing block 36 
synthesizes the input video signal with other data on a 
multimedia plane without scale conversion and outputs the 
synthesized data. 

[0192] Thus, recording the multimedia display sub- 
information and using it at the time of reproduction allow 
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the receiving side to display a screen as intended on the 
sending side. Referring to FIG . 22, if the re-encoding on 
the sending side (recording side) results in a smaller video 
picture frame than the original, the size reduction is 
recorded as multimedia display sub-information, which is 
referenced at the time of reproduction. Consequently, 
because there exists no mismatch between video data and 
other data, the receiving side (the reproduction side) can 
display the same screen as the original. 

[0193] FIG. 24 is a flowchart describing AV stream 
reproduction processing which uses multimedia display sub- 
information . 

[0194] In step 60, a multiplexed stream including 
multimedia coding data is read from a recording medium and 
input in a reproduction device . 

[0195] In step 61, multimedia display sub-information is 
input. This information is read from the recording medium 
in the case of the reproducing device shown in FIG. 21; in 
the case of a reproducing device shown in FIG. 25, this 
information is separated from the multiplexed stream. 
[0196] In step 62, a video stream is separated from the 
multiplexed stream. 

[0197] In step 63, the video stream is decoded. 
[0198] In step S64, if there — exists — a mismatch exists ^-s- 
dioplay — between the video data and the multimedia coding 
data, the synthesizing block 36 scale-converts the video 
data on the basis of the multimedia display sub-information. 
[0199] In step 65, the synthesizing block 36 synthesizes 
the processed image and the multimedia data to generate a 
display image . 

[0200] As described, the multimedia display sub- 
information may be recorded on the recording medium 2^2 as a 
file which is different from the source packet stream file 
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containing character graphics data and video signals. 
Alternatively, the mutlimedia display sub-information may be 
embedded in a source packet stream file and then recorded on 
the recording medium 22. FIG. 23 shows the configuration of 
the recording apparatus 1 in which the multimedia display 
sub-information is embedded in a source packet stream file. 
[0201] In comparison between the configuration of the 
recording apparatus 1 shown in FIG. 23 and the configuration 
shown in FIG. 3, the former outputs the multimedia display 
sub-information output from the coding block 20 and jrs 
suppliesd this output to the multiplexer 16. The 
multiplexer 16 then generates a transport packet of the 
input multimedia display sub-information and embeds it into 
a source packet stream file, outputting the same to the 
arrival timestamp adding block 17. Instead of embedding the 
multimedia display sub-information into a source packet 
stream file as a transport packet, the multimedia display 
sub-information may be written to a user data area in an 
MPEG video stream. 

[0202] In the present embodiment of the invention, video 
data may be re-encoded if* — using other methods than that 
described above; for example, an input video stream may be 
converted in the DCT area to convert the coding parameters 
such as picture frame . 

[0203] FIG. 25 shows the configuration of the reproducing 
apparatus 30 in which the multimedia display sub-information 
is embedded in a source packet stream file to be recorded on 
the recording medium 22. In comparison between the 
configuration of the reproducing apparatus shown in FIG. 25 
and the configuration shown in FIG. 21, the former reads 
only the source packet stream through the reading block 31 
only — the — sourco — packet — stream . The source packet stream 
read by the reading block 31 is input i-n to the 
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demultiplexer 33 via the arrival timestamp separating block 
32. 

[0204] The demultiplexer 33 extracts the multimedia 
display sub-information from the input source packet stream 
file and outputs the extracted information to the 
synthesizing block 36. The further processing is the same 
as that of the configuration shown in FIG. 5. 

[0205] Thus, if the multimedia display sub-information is 
recorded as embedded in a source packet stream file, the 
receiving side can also obtain the video picture size and 
display position intended by the sending side. 

[0206] In the present embodiment of the invention, a 
transport stream was used — as an example. The present 
invention is — also is applicable to multiplexed streams such 
as a program stream. 

[0207] The above-described sequence of processing 
operations can be executed by hardware as well as software. 
In the software approach, the recording apparatus 1 (and the 
reproducing apparatus 30) is constituted by a personal 
computer as shown in FIG. 26. 

[0208] Referring to FIG. 26, a CPU (Central Processing 
Unit) 101 executes various processing operations as 
instructed by programs stored in a ROM (Read Only Memory) 
102 or loaded from a storage block 108 into a RAM (Random 
Access Memory) 103. The RAM 103 also stores, as required, 
the data necessary for the CPU 101 to execute various 
processing operations . 

[0209] The CPU 101, the ROM 102, and the RAM 103 are 
interconnected via a bus 104. The bus 104 4rS — also .is 
connected to an input/output interface 105. 

[0210] The input/output interface 105 is connected to an 
input block 106^ such as a keyboard and a mouse, a display 
device such as a CRT or LCD, an output block 107^ such as a 
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speaker, a storage block 108 such as hard disk, and a 
communication block 109 such as modem or terminal adapter. 
The communication block 109 executes communication 
processing via a network. 

[0211] The input/output interface 105 4rs also is 

connected to a drive 110_^_ as required, in which a magnetic 
disc 121, an optical disc 122, a magneto-optical disc 123, 
or a semiconductor memory 124 is loaded. Computer programs 
read from these storage media are installed in the storage 
block 108 as required. 

[0212] The execution of a sequence of processing 
operations by software requires the use of a computer having 
a dedicated hardware device storing beforehand the programs 
constituting the software or a general-purpose computer in 
which these programs are installed^ as required^ from a 
recording medium. 

[0213] The program recording medium for storing computer- 
readable and executable programs may be a package medium 
which is distributed to users providing programs and 
constituted embodied by the magnetic disk 121 or (including 
floppy disk), the optical disc 122 (including CD-ROM 
(Compact Disc-Read Only Memory) and DVD (Digital Versatile 
Disc) ) , the magneto-optical disk 123 (including MD (Mini 
Disk) ) , or the semiconductor memory 124^_ or a ROM 102 or a 
hard disk which is preinstalled in a personal computer and 
provided for users and on which the programs are stored 
temporarily or permanently as shown in FIG. 26. 
[0214] It should be noted that the steps describing the 
programs to be stored ion the program storage medium are not 
only executed in a time-dependent manner in the order 
described, but also in parallel or in a discrete manner. 
[0215] As described^ and according to the first image 
coding apparatus and method and the program stored in the 
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first recording medium, a video steam is separated from a 
multiplexed stream containing multimedia coding data, a 
predetermined conversion process is performed on the 
separated video stream, and additional information 
indicative that — of a mismatch will — occurs when displaying 
the converted video stream on the basis of the multimedia 
coding data. 

[0216] The first recording medium stores the converted 
video stream, the multimedia coding data, and the additional 
information indicative that a mismatch will occur when 
displaying the converted video stream on the basis of the 
above-mentioned multimedia coding data. 

[0217] Consequently, in any case, the reproducing side 
can prevent a mismatch from occurring between the video 
stream and the multimedia coding data. 

[0218] As described and according to the image decoding 
apparatus and method and the program stored in the second 
recording medium, a mismatch occurs when a video stream is 
separated from an input multiplexed stream, the separated 
video stream is decoded, and the decoded video stream is 
displayed on the basis of multimedia coding information — a 
mismatch occurs . On the basis of the additional information 
about this mismatch occurrence, a predetermined conversion 
process is performed on the decoded video stream. This 
novel configuration prevents the mismatch from occurring 
between the video stream and the multimedia coding data. 
[0219] As described and according to the second image 
coding apparatus and method and the program stored in the 
third recording medium, a video stream is separated from an 
input multiplexed stream, the input multiplexed stream is 
checked for multimedia coding data and, if the multimedia 
coding data is found, coding control information for giving 
an instruction not to change the display format of the 
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separated video stream is generated, and a predetermined 
conversion process is performed on the separated video 
stream on the basis of the generated coding control 
information . 

[0220] The second recording medium also stores the above- 
mentioned coding control information giving instruction not 
to change the display format of a video stream and a 
multiplexed stream containing the video stream on which a 

predetermined conversion process has been performed on the 

i 

basis of the coding control information. 

[0221] Consequently, in any case, the reproduction side 
can prevent a mismatch from occurring between the video 
stream and the multimedia coding data. 

[0222] Although the invention herein has been described 
with reference to particular embodiments, it is to be 
understood that these embodiments are merely illustrative of 
the principles and applications of the present invention. 
It is therefore to be understood that numerous modifications 
may be made to the illustrative embodiments and that other 
arrangements may be devised without departing from the 
spirit and scope of the present invention as defined by the 
appended claims. 
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